Papers with multimodal understanding
InfiMM: Advancing Multimodal Understanding with an Open-Sourced Visual Language Model (2024.findings-acl)
Copied to clipboard
Haogeng Liu, Quanzeng You, Yiqi Wang, Xiaotian Han, Bohan Zhai, Yongfei Liu, Wentao Chen, Yiren Jian, Yunzhe Tao, Jianbo Yuan, Ran He, Hongxia Yang
| Challenge: | InfiMM is a multimodal large language model that adapts to complex vision-language tasks. |
| Approach: | They present a Multimodal Large Language Model that adapts to intricate vision-language tasks using large-scale training data and comprehensive training strategies. |
| Outcome: | Empirical evaluations across a variety of benchmarks underscore InfiMM’s remarkable capability in multimodal understanding. |
Prompt Tuning for Unified Multimodal Pretrained Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Prompt tuning has demonstrated success in natural language pretraining and even vision pretraining. |
| Approach: | They propose to apply prompt tuning to a unified sequence-to-sequence pretrained model by adding a sequence of learnable embeddings to each layer and finetuning the pretrained models on downstream tasks. |
| Outcome: | The proposed method outperforms other parameter-efficient tuning methods on multimodal models and is robust against adversarial attacks. |
Multimodal Intent Discovery from Livestream Videos (2022.findings-naacl)
Copied to clipboard
Adyasha Maharana, Quan Tran, Franck Dernoncourt, Seunghyun Yoon, Trung Bui, Walter Chang, Mohit Bansal
| Challenge: | Existing models for instructional video understanding struggle to understand abstract intents . identifying procedural intent within instructional videos is a challenging task . |
| Approach: | They propose to extract instructional intent from software instructional livestreams by using a multimodal cascaded cross-attention model that integrates weaker and noisier video signals with more discriminative text signals. |
| Outcome: | The proposed model improves on baseline models and compares it to existing models. |
Thesis Proposal: An Explainable Multimodal Framework for Detecting Harmful Content in Code-Switched Children’s Media (2026.acl-srw)
Copied to clipboard
| Challenge: | Current content moderation systems fail to protect children from harmful content, especially in under-resourced, code-switched settings. |
| Approach: | They propose to integrate a fine-tuned classifier with an LLM-powered module that synthesizes the classifier’s internal evidential signals to generate faithful, human-readable rationales for each decision. |
| Outcome: | The proposed framework integrates a fine-tuned classifier for accurate, scalable detection with an LLM-powered module that synthesizes the classifier’s internal evidential signals to generate faithful, human-readable rationales for each decision. |
Hospitality-VQA: Decision-Oriented Informativeness Evaluation for Vision–Language Models (2026.eacl-srw)
Copied to clipboard
Jeongwoo Lee, Baek Duhyeong, Eungyeol Han, Soyeon Shin, Gukin Han, Seungduk Kim, Jaehyun Jeon, Taewoo Jeong
| Challenge: | Existing VQA benchmarks focus on factual correctness but rarely capture what information users actually find useful. |
| Approach: | They propose a framework to quantify how much information an image–question pair provides . they conduct experiments with several state-of-the-art VLMs to determine their reliability . |
| Outcome: | The proposed framework quantifies how much information an image–question pair provides in hospitality contexts. |
Data-Centric Improvements for Enhancing Multi-Modal Understanding in Spoken Conversation Modeling (2025.findings-acl)
Copied to clipboard
| Challenge: | Conversational assistants are increasingly popular across diverse real-world applications . speech data constitute high-dimensional signals that are difficult to model even for frontier models . |
| Approach: | They propose a data-centric customization approach for enhancing multimodal understanding in conversational speech modeling. |
| Outcome: | The proposed model achieves state-of-the-art on the Spoken-SQuAD benchmark using 10% of training data with open-weight models. |
EasyGen: Easing Multimodal Generation with BiDiffuser and LLMs (2024.acl-long)
Copied to clipboard
| Challenge: | Existing multimodal models that depend on encoders like CLIP or ImageBind need ample amounts of training data to bridge modalities. |
| Approach: | They propose an efficient model that leverages bidirectional conditional diffusion model to foster more efficient modality interactions. |
| Outcome: | The proposed model is able to train a projection layer linking an LLM and an adapter to align the LLM’s text space with the bidirectional diffusion model. |
Speaking Beyond Language: A Large-Scale Multimodal Dataset for Learning Nonverbal Cues from Video-Grounded Dialogues (2025.acl-long)
Copied to clipboard
Youngmin Kim, Jiwan Chung, Jisoo Kim, Sunghyun Lee, Sangkyu Lee, Junhyeok Kim, Cheoljong Yang, Youngjae Yu
| Challenge: | Existing large language models fail to incorporate nonverbal elements into conversational experiences. |
| Approach: | They propose a multimodal language model that generates nonverbal cues alongside text . their dataset is annotated with time-aligned text, facial expressions, and body language . |
| Outcome: | The proposed model generates nonverbal languages and text, corresponding to conversational input. |
PreGenie: An Agentic Framework for High-quality Visual Presentation Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Visual presentations are vital for effective communication, but they are limited by their complexity and lack of visual understanding. |
| Approach: | a new framework is proposed to generate high-quality visual presentations using multimodal large language models. |
| Outcome: | The proposed framework outperforms existing models in multimodal understanding and content consistency. |
Can Language Models Laugh at YouTube Short-form Videos? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets that focus on verbal cues and focus on short-form funny videos focus on focusing on verbs and visual cue. |
| Approach: | They curate a user-generated dataset of 10K multimodal funny videos from YouTube and annotate each video with timestamps and explanations for funny moments. |
| Outcome: | The proposed dataset improves the ability of large language models to understand humor. |
Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering (2025.naacl-long)
Copied to clipboard
| Challenge: | Generative AI has made rapid advances in multimodal understanding and code generation. |
| Approach: | They construct a first real-world benchmark for multimodal large language models that directly convert visual designs into code implementations by manually curating 484 diverse real-life webpages as test cases. |
| Outcome: | The proposed model can generate code implementations that directly render into the given reference webpages, given the screenshots as input. |
AdaV: Adaptive Text-visual Redirection for Vision-Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Vision-language models often generate excessive visual tokens, leading to poor performance . a novel training-free visual token pruning method is proposed to improve performance despite the computational cost associated with VLMs. |
| Approach: | They propose a training-free visual token pruning method that reduces biased token pruning . they plan to open-source the code upon publication . |
| Outcome: | The proposed method reduces biased token pruning and enhances model robustness with limited visual token budget. |
“I See What You Did There”: Can Large Vision-Language Models Understand Multimodal Puns? (2026.acl-long)
Copied to clipboard
Naen Xu, Jiayi Sheng, Changjiang Li, Chunyi Zhou, Yuyuan Li, Tianyu Du, Jun Wang, Zhihui Fu, Jinbao Li, Shouling Ji
| Challenge: | Puns are a common form of rhetorical wordplay that exploits polysemy and phonetic similarity to create humor. |
| Approach: | They propose a multimodal pun generation pipeline and a model to evaluate their understanding of puns. |
| Outcome: | The proposed benchmark improves the understanding of multimodal puns by 16.5% in the F1 test. |
RRHF-V: Ranking Responses to Mitigate Hallucinations in Multimodal Large Language Models with Human Feedback (2025.coling-main)
Copied to clipboard
| Challenge: | Existing methods to mitigate hallucinations generate erroneous or fabricated information. |
| Approach: | They propose a rank-response-based model that annotates pair-reponses and trains alignment algorithms to improve the correspondence between images and text. |
| Outcome: | The proposed model outperforms the DPO method and outperfies existing methods on two MLLMs of different sizes and four widely used benchmarks. |
Beyond Logit Lens: Contextual Embeddings for Robust Hallucination Detection & Grounding in VLMs (2025.naacl-long)
Copied to clipboard
Anirudh Phukan, Divyansh Divyansh, Harshit Kumar Morj, Vaishnavi Vaishnavi, Apoorv Saxena, Koustava Goswami
| Challenge: | Large Multimodal Models are plagued by hallucinations that limit their reliability and adoption. |
| Approach: | They propose a method that leverages contextual token embeddings from LMMs to detect hallucinations. |
| Outcome: | The proposed method improves hallucination detection and grounding across diverse categories while excelling in tasks requiring contextual understanding. |
Imagination and Contemplation: A Balanced Framework for Semantic-Augmented Multimodal Machine Translation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Multimodal Machine Translation (MMT) is effective in resolving linguistic ambiguities, but visual information often introduces redundancy or noise, potentially impairing translation quality. |
| Approach: | They propose a semantic-augmented framework that integrates "Imagination" and "Contemplation" they first generate synthetic images from source text and align them with authentic images via an optimal transport loss . |
| Outcome: | The proposed framework outperforms baselines on translation datasets with visually ambiguous or weakly correlated content. |
Activation Steering Decoding: Mitigating Hallucination in Large Vision-Language Models through Bidirectional Hidden State Intervention (2025.acl-long)
Copied to clipboard
| Challenge: | Large Vision Language Models (LVLMs) suffer from hallucination where generated textual descriptions fail to align accurately with visual semantics. |
| Approach: | They propose a training-free approach that mitigates hallucination through targeted intervention in the model’s intermediate activations by identifying directional patterns of hallucinism in the activation space using a small calibration set. |
| Outcome: | The proposed approach reduces hallucination across multiple benchmarks while maintaining performance on general visual understanding tasks. |
BcQLM: Efficient Vision-Language Understanding with Distilled Q-Gated Cross-Modal Fusion (2025.findings-emnlp)
Copied to clipboard
| Challenge: | proposed lightweight MLLM framework for end-to-end visual question answering . proposed framework uses BreezeCLIP, a vision-language encoder optimised for efficient multimodal understanding . |
| Approach: | proposed lightweight MLLM framework is based on BreezeCLIP, a vision-language encoder . it offers a promising path toward deployable ML models under practical hardware constraints. |
| Outcome: | The proposed model significantly reduces computational cost while achieving performance comparable to standard-size MLLMs. |
Game on Tree: Visual Hallucination Mitigation via Coarse-to-Fine View Tree and Game Theory (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large vision-language models produce unfaithful visual hallucinations, also known as visual halluinations, which hinders their application in multimodal understanding and decision-making. |
| Approach: | They propose a plug-and-play train-free decoding algorithm for mitigating visual hallucinations . they leverage visual information to construct a coarse-to-fine visual view tree . |
| Outcome: | The proposed algorithm reduces visual hallucinations (VH) by leveraging visual information to construct a coarse-to-fine visual view tree (CFTree) |
Datasets for Scientific Literature Understanding: A Survey (2026.findings-acl)
Copied to clipboard
| Challenge: | Empowering machines to understand scientific literature is crucial for accelerating scientific discovery and advancing the AI for Science paradigm. |
| Approach: | They propose a systematic taxonomy that organizes resources spanning structural understanding, text understanding, multimodal understanding and pre-training/instruction fine-tuning. |
| Outcome: | The proposed taxonomy organizes resources spanning structural understanding, text understanding, multimodal understanding and pre-training/instruction fine-tuning. |
MULTIVOX: A Benchmark for Evaluating Voice Assistants for Multimodal Interactions (2025.emnlp-main)
Copied to clipboard
Ramaneswaran Selvakumar, Ashish Seth, Nishit Anand, Utkarsh Tyagi, Sonal Kumar, Sreyan Ghosh, Dinesh Manocha
| Challenge: | omni models lack spoken dialogues, which is essential for assessing conversational and auditory capabilities of voice assistants. |
| Approach: | They propose a benchmark to evaluate the ability of voice assistants to integrate paralinguistic speech features into their models. |
| Outcome: | The multivox voice assistant benchmark evaluates the ability of models to integrate spoken and visual cues including paralinguistic speech features for truly multimodal understanding. |
UnifiedVisual: A Framework for Constructing Unified Vision-Language Datasets (2025.emnlp-main)
Copied to clipboard
Pengyu Wang, Shaojun Zhou, Chenkun Tan, Xinghao Wang, Wei Huang, Zhen Ye, Zhaowei Li, Botian Jiang, Dong Zhang, Xipeng Qiu
| Challenge: | Existing datasets address understanding and generation in isolation, limiting the performance of unified vision large language models. |
| Approach: | They propose a dataset that facilitates mutual enhancement between multimodal understanding and generation. |
| Outcome: | The proposed framework integrates diverse visual and textual inputs and outputs, enabling comprehensive cross-modal reasoning and precise text-to-image alignment. |
Taking Notes Brings Focus? Towards Multi-Turn Multimodal Dialogue Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing multimodal large language models are trained on single-turn vision question-answering tasks, which do not accurately reflect real-world human conversations. |
| Approach: | They propose a large-scale multi-turn multimodal dialogue dataset that uses rules and GPT assistance to generate a multi-turned multimodal dialog dataset. |
| Outcome: | The proposed dataset is a strong benchmark for multi-turn multimodal dialogue learning . it features complex dialogues with contextual dependencies that force models to track, ground, and recall information across multiple turns and disparate visual regions. |
M3-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering (2026.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge-based VQA benchmarks focus on coarse-grained categories and simple reasoning over single entities. |
| Approach: | They propose a knowledge-based Visual Question Answering benchmark to enhance multimodality evaluation. |
| Outcome: | The proposed benchmark improves evaluation of multimodal large language models in fine-grained multimodal entity understanding and complex multihop reasoning. |
Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents (2026.acl-long)
Copied to clipboard
Yuanchen Bei, Tianxin Wei, Xuying Ning, Yanjun Zhao, Zhining Liu, Xiao Lin, Yada Zhu, Hendrik Hamann, Jingrui He, Hanghang Tong
| Challenge: | Existing benchmarks evaluate multi-session memory in text-only conversations or assess multimodal understanding within localized contexts. |
| Approach: | They propose a benchmark for evaluating multimodal long-term conversational memory in MLLM agents. |
| Outcome: | The proposed framework assesses key memory capabilities along three functional dimensions: memory extraction and test-time adaptation, memory reasoning, and memory knowledge management. |
Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understanding (2026.acl-long)
Copied to clipboard
Jaehyun Jeon, Min Soo Kim, Janghan Yoon, Sumin Shim, Yejin Choi, Hanbin Kim, Dae Hyun Kim, Youngjae Yu
| Challenge: | Recent studies focus on surface-level features, overlooking how design choices influence user behavior at scale. |
| Approach: | They propose a benchmark for multimodal understanding of how UI/UX design affects user behavior built on 300 real-world UI image pairs from industry A/B tests. |
| Outcome: | The proposed benchmarks show that models exhibit limited understanding of the behavioral impact of UI/UX design. |
Rethinking Composed Image Retrieval Evaluation: A Fine-Grained Benchmark from Image Editing (2026.acl-long)
Copied to clipboard
Tingyu Song, Yanzhao Zhang, Mingxin Li, Zhuoning Guo, Dingkun Long, Pengjun Xie, Siyue Zhang, Yilun Zhao, Shu Wu
| Challenge: | Composed Image Retrieval (CIR) is a complex task in multimodal understanding . current CIR benchmarks lack a robust evaluation pipeline and limited query categories . |
| Approach: | They construct a fine-grained CIR benchmark that allows for precise control over modification types and content. |
| Outcome: | The proposed benchmark covers 5,000 high-quality queries structured across five main categories and fifteen subcategories. |